Introducing ‘annolite’. A Leightweight Fulltext Display and Annotation Toolset

The fulltextlist class

The S3 class fulltextlist is a list of lists with information on the content and the formatting of the chunks of text that make up a document. The class and its data structure are designed such that it can be generated from any kind of text input and transferred seamlessly to JavaScript. This is why fulltext is represented by a simple S3 class.

The lists that make up the fulltextlist class are the chunks of text that make up a text documment. Each chunk is a list with the following building blocks:

• element: A length-one character vector defining the html element that will define the layout of a chunk of text when it is assembled by the htmlwidget (e.g. “h1” for level 1 headline, “p” or “para” for plain paragraphs)

• attributes: A named character vector defining attributes of the element and the values of the attributes. Attributes used at this stage are “style” with the values “display:block” or “display:none”, and “name”. The attribute “name” is relevant when multiple documents are displayed in one combined HTML element.

• tokenstream: The actual text of each chunk of text (a headline, a paragraph, or any other region of text) is represented by a data.frame in a tokenized, tabular format. This data.frame needs to include a column id with a unique token id (used to match annotations and tokens), a colum token with the wordform to be displayed, and a column whitespace that includes the whitespace to be prepended to a token.

The fulltextlist() method serves as a the constructor for the fulltextlist class. It is flexible and will facilitate working with annolite without strong assumptions on the data structure you should offer. The method accepts different inputs. The most elementary input is a list of character vectors.

Jane Austen’s novel Emma is a popular text for teaching text analysis with R. It is part of the R package janeaustenr and is part of the examples used for the tidy approach to text mining using the (see Julia Silge and David Robinson’s book Text Mining with R). This package includes the book as sample data, a list of the chapters of Emma that are lists of character vectors representing tokenized paragraphs. The data looks as follows.

emma_chapters_tokenized[[1]][[1]]

##  [1] "Emma"        "Woodhouse"   ","           "handsome"    ","          
##  [6] "clever"      ","           "and"         "rich"        ","          
## [11] "with"        "a"           "comfortable" "home"        "and"        
## [16] "happy"       "disposition" ","           "seemed"      "to"         
## [21] "unite"       "some"        "of"          "the"         "best"       
## [26] "blessings"   "of"          "existence"   ";"           "and"        
## [31] "had"         "lived"       "nearly"      "twenty"      "-"          
## [36] "one"         "years"       "in"          "the"         "world"      
## [41] "with"        "very"        "little"      "to"          "distress"   
## [46] "or"          "vex"         "her"         "."

To turn the first chapter of emma into a fulltextlist, supply the fulltextlist() method with the list of character vectors with the tokenized paragraphs as input.

emma_ch1 <- fulltexttable(emma_chapters_tokenized[[1]])

The object emma_ch1 is the list of chunks introduced before. To illustrate, we inspect the first chunk.

emma_ch1[1,]

##       name element         style
## (3,8]         para display:block
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   tokenstream
## (3,8] Emma, Woodhouse, ,, handsome, ,, clever, ,, and, rich, ,, with, a, comfortable, home, and, happy, disposition, ,, seemed, to, unite, some, of, the, best, blessings, of, existence, ;, and, had, lived, nearly, twenty, -, one, years, in, the, world, with, very, little, to, distress, or, vex, her, ., -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, ,  , ,  , ,  , ,  ,  , ,  ,  ,  ,  ,  ,  ,  , ,  ,  ,  ,  ,  ,  ,  ,  ,  ,  , ,  ,  ,  ,  ,  ,  ,  ,  ,  ,  ,  ,  ,  ,  ,  ,  ,  ,  ,  ,

Having the fulltextlist object is enough to generate a htmlwidget that

annolite(emma_ch1)

To learn more about the using the fulltextlist() method and its flexibility, consult the documentation (?fulltextlist).

The functionality to include fulltext based on the fulltextlist class in an Rmarkdown may be convenient at times. But annolite is about annotating text and this requires a furter data structure to manage annotations. This is the purpose of the the annotationstable class.

The annotationstable class

The annotationstable class is a S3 superclass of a data.frame. So it is and behaves like a data.frame, but using the class is useful for being able to define specified methods. An annotationsstable has the columns “text”, “code”, “color”, “annotation” , “start” and “end”. This captures all the information generated when annotating a document using annolite.

To convey what this table looks like, inspect the sample_annotation object that is included as sample annotation in the package.

sample_annotation

##                                                                                             text
## 1                                                              eine sehr verehrten Damen und Her
## 2                                                                    wierigsten Finanzund Wirtsc
## 3                                                                            Regierungskoalition
## 4 rbeitsplätze sorgt . Wir haben uns sehr intensiv Gedanken darüber gemacht , wie wir auch in sc
##     code  color annotation start   end
## 1 orange orange       Asdf 36137 36142
## 2 orange orange        Uff 36175 36177
## 3 orange orange        Gov 36260 36260
## 4 orange orange    labor\n 36418 36434

Note that what you see in the column “text” is the text selection when creating an annotation that does not necessarily cover entire words. The token ids in this table are fairly high numbers because these are (arbitrary) annotations of a speech given by Volker Kauder as part of the GermaParl - a fairly small subset of a much larger corpus.

To learn more about the annotationstable, see the documentation of the annotationstable() function (?annotationstable) that serves as a constructor of an (empty) annotationstable.

Inspecting multiple annotated documents

Annotation will rarely be limited to a single document. Usually, a larger collection of documents will be annotated. It would be tedious having to write the code to prepare the htmlwidget for every single document. Yet you can combine multiple htmlwidgets into one browsable HTML element that can communicate with each other via a mechanism introduced with the crosstalk package. The annolite htmlwidget is crosstalk-enabled and provides the possibility to inspect multiple annotated documents in one HTML element.

Technically, the HTML element you can use to inspect multiple annotated documents will consist of a datatables htmlwidget as created by DT::datatable(), and an annolite htmlwidget. The table will list the names of the documents. By clicking a document name, the respective annotated documment will be displayed.

The annolite() method will prepare this output. But this time, the input needs to be a list of fulltextlist objects with defined names. We use the chapters of emma as an example and start with preparing the list of fulltextlist objects. Each of these objects in the list will include the fulltext of a chapter.

emma_chapters <- lapply(emma_chapters_tokenized, fulltexttable)

The interaction between the table and the annolite htmlwidget requires that the name compontent is a non-empty character vector. So we iterate through the chapters again and fill the name component of the fulltextlist objects.

emma_chapters <- lapply(
  seq_along(emma_chapters),
  function(i){
    emma_chapters[[i]][["name"]] <- sprintf("Chapter %s", i)
    emma_chapters[[i]]
  }
)

This is an example where you fill find the name …

unique(emma_chapters[[2]][["name"]])

## [1] "Chapter 2"

The input required for an html element with a table to select documents and a documeent viewer is fully prepared. In this example, we limit the input to the first five chapters of Emma to be somewhat parsimonious with the data to be included in the vignette.

annolite(do.call(rbind, emma_chapters), group = "emma")

Evaluating Topic Models

The original intention behind the development of annolite is to offer a leightweight open source tool for a pure R workflow to create and display annotations. The widget is sufficiently flexible to allow alternative uses. You can also use it as a viewer to highlight tokens or token sequences, any kind of dictionary that is relevant for your work.

The display mode of the annolite widget can be used to qualitatively evaluate how a topic model works. In Blei’s original work on topic models, there is a very telling snippet of a text highlighting the words that indicate three different topics (Blei 2012, 78).

The annolite widget can get you this kind of output which may be very useful to evaluate topic models. In the example we use to exemplify this usage, a topic model (k = 180) is computed for the speeches in the United Nations General Assembly. Topic 105 might be indicative for a thematic focus on international migration. To evaluate this, the following HTML element can be used - you can skip through the speeches that went into the computation of the topicmodel, the terms indicative for topic 105 (first 50 terms only) are highlighted in yellow. Note that to maintain a reasonable size of the vignette, only 25 documents are included here.

annolite(
  x = unga_migrationspeeches_fulltext,
  annotations = unga_migrationspeeches_anntationstable,
  group = "unga"
)

To limit the number of dependencies, to avoid having to download somewhat large data when preparing the vignette, and to limit computation time, the objects unga_migrationspeeches_fulltext and unga_migrationspeeches_anntationstable are included as sample data in the package. The code used is in the “data-raw” folder.

Using the document viewer in slides

Including an annolite htmlwidget as such, or an HTML element that integrates it into a HTML presentation may be a great way to explain or document research. However, generating a an ioslides presentation from Rmarkdown (Xie, Allaire, and Grolemund 2018), which is flexible and comfortable in general, will usually entail some awkward CSS conflicts when it comes to including an annolite htmlwidget. The best practice we could identify is to wrap the HTML element that includes an annolite htmlwidget into a “widgetframe”, an approach introduced with the widgetframe package. Technically, the htmlwidget will be put into an iframe, thus containing it and avoding CSS conflicts.

A template for creating slides included in the package provides further explanations and includes the relevant code. The template is available from RStudio via the menu: File > New File > R Markdown > From Template > slides {annolite}.